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Abstract 

The entropy of an orthogonal matrix is defined. It provides a new inter- 
pretation of Hadamard matrices as those that saturate the bound for entropy. 
It appears to be a useful Morse function on the group manifold. It has sharp 
maxima and other saddle points. The matrices corresponding to the max- 
ima for 3 and 5 dimensions are presented. They are integer matrices (upto a 
rescaling.) 
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Hadamard's maximum determinant problem asks for nxn matrix h n with 
elements of magnitude < 1 which has the maximum possible magnitude for 
the determinant. Hadamard's inequality is |det(/i n )| < . The equality 
is attained for some values of n, and the corresponding matrices are called 
Hadamard matrices H n . They are orthogonal matrices upto a proportionality 
factor \fn: 



and have elements ±1. Geometrical interpretation of the maximum deter- 
minant problem is to look for n vectors from the origin contained within 
the cubes \xi\ < 1, % = l,..,n and forming a rectangular parallelopiped of 
maximum volume. In case of Hadamard matrices, these vectors are a set of 
main diagonals, which obviously have the maximum possible lengths. More- 
over they form an orthogonal set to provide the maximum volume. In other 
dimensions, it is not possible to have mutually orthogonal main diagonals. 
Nevertheless, we expect that the maximum determinant is from vectors which 
are 'close' to some main diagonals. This means that the entries of the matrix 
have magnitudes close to 1. Also we might expect a continuous set of matri- 
ces which have the maximum determinant because certain allowed changes 
in the vectors would increase their lengths and decrease the angles between 
them keeping the volume unchanged. 

Hadamard matrices find natural applications in error-correcting codes [1] 
and provide "biorthogonal" codes. There is extensive work on finding matri- 
ces with entries ±1 which have maximum determinant. This is a subclass of 
the Hadmard's problem. There are also many generalizations and relatives of 
the problem. For example, [2] studies the computationally difficult problem 
of finding a largest j-dimensional simplex in a given d- dimensional cube. 

In this note we use entropy to give a new criterion for Hadamard ma- 
trix and obtain its generalization. We define entropy of orthogonal matrices 
and Hadamard matrices (appropriately normalized) saturate the bound for 
the maximum of the entropy. The maxima (and other saddle points) of 
the entropy function have intriguing structure and yield generalizations of 
Hadamard matrices. It appears that entropy is a very useful Morse function 
on the group manifold and not only the number [3] but also the location of 
the saddle points have interesting features. 



H n Hl = nl, 
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Consider n random variables with a set of possible outcomes i = l,..,n 

n 

having probabilities Pi, i = l,..,n. We have pj — 1. The (Shannon) 

i=i 

entropy is 

n i 

^fe} = E^ ln - (2) 

i=1 Pi 

This has the minimum value zero for the case of certainty, 

Pi = 



1 i = j for some j, 
i ^j. 



It has the maximum value In n when all outcomes are equally likely, 

Pi = - , Vi = 1, 
n 

We now define entropy of an orthogonal matrix O 1 -, z,j = 1, 2,...n. 
Here O l -'s are real numbers with the constraint 

to 1 ,01 = S jk . (3) 

i=l 

In particular, row of the matrix is a normalized vector for each i 

of = (O*, 

,(*) 



1, We may associate probabilities = (O 1 ) 2 with the i th row, as 



pj = 1, for each z. We define the (Shannon) entropy for the orthogonal 

3=1 

matrix as the sum of the entropies of each row: 



h{o i 3 } = - E (o l 3 ) 2 Ho 1 



yi \2 
i,3=l 



The minimum value zero is attained by the identity matrix O l j = 5j and 
related matrices obtained by interchanging rows or changing the signs of the 
elements. The entropy of the i th row can have the maximum value Inn, 
which is attained when each element of the row is ±-J=. This gives the 
bound, H{O l j} < n Inn. In general, the entropy of an orthogonal matrix 
cannot attain this bound because of the orthogonality constraint (3) which 
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constrains pj for different rows. In fact the bound is obtained only by the 
Hadamard matrices (rescaled by n~^). Thus we have a new criterion for the 
Hadamard matrices (appropriately normalized): those orthogonal matrices 
which saturate the bound for entropy. 

Note that the entropy is large when each element is as close to ±^ 
as possible, i.e., to a main diagonal. Thus maximum entropy condition is 
similar to the maximum determinant condition of the Hadamard. Moreover 
we find that the peaks of entropy are isolated and sharp in contrast to the 
determinant. Also, the matrices corresponding to the maxima have very 
interesting features even for those dimensions n for which Hadamard matrices 
do not exist. We have obtained matrices maximizing the entropy for n = 3 
and 5 by numerical computation. For n = 3, the matrix is 

" _I 2 2 

3 3 3 

2 _ 1 2 

3 3 3 

2 2 _ 1 

3 3 3 

The matrix entries are all rational numbers and with a rescaling of each row 
by 3 we get an integer matrix. This was unexpected. For n = 5, the result is 
similar: the magnitudes of the elements in each row are | repeated 4 times 
and a |. This set can be generalized for any n: 

n -^, 2 -,-, 2 - (5) 

n n n 

(n-i)times 

A vector with these components has unit normalization as a consequence of 
the identity, 

n 2 = (n - 2) 2 + 2 2 (n - 1) 

We expect that for each n > 2, there exist orthogonal matrices with entries 
of magnitude | repeated (n — 1) times along with one in each row. Only 
the location of the element and the signs of the elements change from 
row to row. Nonetheless it is easy to argue that this cannot be the maximum 
for all n. For n very large the rows are close to one of the axes, and not to a 
main diagonal. Therefore the entropy is close to zero instead of being close to 



(4) 
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the bound n In n. We expect that different maxima emerge as n increases. It 
will be interesting to find the matrix corresponding to the maximum entropy 
for large n, but it is numerically difficult. 

Indeed, for n — 4, the Hadamard matrix and not the family (4) gives 
the maximum. Numerical computation has shown that even in this case, 
member of the family (5) is an extremum of the entropy, though not the 
maximum. 

The picture that emerges is as follows. For each n, there are saddle points 
apart from maxima and minima. For example, for n = 3 there is a saddle 
point and the corresponding matrix is: 



1 

2 


1 


1 ' 

2 




1 

V2 





1 


(6) 


1 

2 


1 

V2 


1 

2 





The entropy is peaked quite sharply at all extrema. As we change n, new 
maxima may emerge as for n — 4. There will also be crossovers among the 
families of saddle points. 

Thus entropy function has a rich set of sharp extrema. It appears to 
be a useful Morse function for the 0(n) group manifold. (It is possible to 
define the corresponding object for other group manifolds too.) Even the 
location of the saddle points i.e., the structure of the corresponding matrix 
is interesting. To our knowledge entropy has not been applied as a Morse 
function in the past. (It has been applied for a classification of the Bernoulli 
shifts [4].) 

The specific way in which the matrices (4), (6) solve the equations for the 
extrema is interesting. It turns out that they are simultaneously extrema of 
a class of entropy functions called Renyi entropy: 

Hai&j} = E((^0 2 ) a 

where a > 0. We can recover the Shannon entropy from 0(e) terms when 
a = 1 — e. To obtain the equations for the extrema we use a Lagrange 
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multiplier A^- symmetric in i and j for the constraint (1). The equations are 

a k 

Using the orthogonality relation (1) we may rewrite this as 

i a 

For j = I, this simply gives the value of Xjj. The nontrivial equations are 
obtained by using Xji = Ay, j ^ I: 

E((oy 2 ) a - 1 o\ o\ = Y,((° i if) a ~ 1 °\ ° l 3 >i*j- 

i i 



We demonstrate how these nonlinear equations are satisfied by the maximum 
(4) for n — 3. Using the elements in the first and second rows, we have, 




We see that either O l j = rkO l l so that the contribution to the left hand side 
and the right hand side in equation (7) are same; or else, O l ■ = ±O l \ for 
some % and i' so that again the contribution to the left hand side and the 
right hand side are the same. 

In this note we defined entropy of orthogonal matrices. We observed 
that it provides a new characterization of Hadamard matrices as those that 
saturate the bound for entropy. In dimensions where Hadamard matrices do 
not exist, entropy has sharp maxima (and other saddle points) in contrast 
to the determinant considered by Hadamard. We obtained matrices that 
maximze entropy in 3 and 5 dimensions. Surprisingly, they have rational 
entries. We argued that as n is varied there are families of maxima and 
saddle points which have interesting cross overs. It is challenging to find the 
maxima for large n (for non-Hadamard dimensions.) We expect it will be 
close to Hadamard matrices in structure. 
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